{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $$User\\ Defined\\ Metrics\\ Tutorial$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/custom_loss/custom_loss_and_metric_tutorial.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents\n",
    "* [1. Introduction](#1.\\-Introduction)\n",
    "* [2. Classification](#2.\\-Classification)\n",
    "* [3. Regression](#3.\\-Regression)\n",
    "* [4. Multiclassification](#4.\\-Multiclassification)\n",
    "* [5. Multiregression](#5.\\-Multiregression)\n",
    "* [6. MultiLabel Classification](#6.\\-MultiLabel-Classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CatBoost allows you to create and pass to model your own loss functions and metrics. To do this you should implement classes with special interfaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\"><b>Tip:</b> Install <code>numba</code> package to speed up computation of your custom losses/metrics. CatBosst will import it for you and perform a just-in-time compilation, if your code is supported by <code>numba</code>. <a href=\"https://numba.pydata.org/numba-doc/dev/reference/pysupported.html\">Here</a> you can check supported Python features.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Interface for user defined objectives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class UserDefinedObjective(object):\n",
    "    def calc_ders_range(self, approxes, targets, weights):\n",
    "        \"\"\"\n",
    "        Computes first and second derivative of the loss function \n",
    "        with respect to the predicted value for each object.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        approxes : indexed container of floats\n",
    "            Current predictions for each object.\n",
    "\n",
    "        targets : indexed container of floats\n",
    "            Target values you provided with the dataset.\n",
    "\n",
    "        weight : float, optional (default=None)\n",
    "            Instance weight.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            der1 : list-like object of float\n",
    "            der2 : list-like object of float\n",
    "\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "class UserDefinedMultiClassObjective(object):\n",
    "    def calc_ders_multi(self, approxes, target, weight):\n",
    "        \"\"\"\n",
    "        Computes first derivative and Hessian matrix of the loss function \n",
    "        with respect to the predicted value for each dimension.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        approxes : indexed container of floats\n",
    "            Predictions for each dimension of single object.\n",
    "\n",
    "        targets : single expected value\n",
    "            True label.\n",
    "\n",
    "        weight : float, optional (default=None)\n",
    "            Instance weight.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            der1 : list-like object of float\n",
    "            der2 : list of lists of float\n",
    "\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "class MultiTargetCustomObjective:\n",
    "    def calc_ders_multi(self, approxes, targets, weights):\n",
    "        \"\"\"\n",
    "        Computes first derivative and Hessian matrix of the loss function \n",
    "        with respect to the predicted value for each dimension.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        approxes : indexed container of floats\n",
    "            Vector of approx labels.\n",
    "\n",
    "        targets : list of float\n",
    "            Vector of true labels.\n",
    "\n",
    "        weight : float, optional (default=None)\n",
    "            Instance weight.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            der1 : list of float\n",
    "            der2 : list of lists of float\n",
    "\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Interface for user defined metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class UserDefinedMetric(object):\n",
    "    def is_max_optimal(self):\n",
    "        \"\"\"\n",
    "        Returns whether great values of metric are better\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        \"\"\"\n",
    "        Evaluates metric value.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        approxes : list of indexed containers (containers with only __len__ and __getitem__ defined) of float\n",
    "            Vectors of approx labels.\n",
    "\n",
    "        targets : one dimensional indexed container of float\n",
    "            Vectors of true labels.\n",
    "\n",
    "        weights : one dimensional indexed container of float, optional (default=None)\n",
    "            Weight for each instance.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            weighted error : float\n",
    "            total weight : float\n",
    "\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "    def get_final_error(self, error, weight):\n",
    "        \"\"\"\n",
    "        Returns final value of metric based on error and weight.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        error : float\n",
    "            Sum of errors in all instances.\n",
    "\n",
    "        weight : float\n",
    "            Sum of weights of all instances.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        metric value : float\n",
    "\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "\n",
    "class MultiTargetCustomMetric:\n",
    "    def evaluate(self, approxes, targets, weights):\n",
    "        \"\"\"\n",
    "        Evaluates metric value.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        approxes : list of lists of float\n",
    "            Vectors of approx labels.\n",
    "\n",
    "        targets : list of lists of float\n",
    "            Vectors of true labels.\n",
    "\n",
    "        weights : list of float, optional (default=None)\n",
    "            Weight for each instance.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            weighted error : float\n",
    "            total weight : float\n",
    "\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        \"\"\"\n",
    "        Returns whether great values of metric are better\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def get_final_error(self, error, weight):\n",
    "        \"\"\"\n",
    "        Returns final value of metric based on error and weight.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        error : float\n",
    "            Sum of errors in all instances.\n",
    "\n",
    "        weight : float\n",
    "            Sum of weights of all instances.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        metric value : float\n",
    "\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we consider examples of user defined metrics for different types of tasks. We will use the following variables:\n",
    "<center>$a$ - approx value</center>\n",
    "<center>$p$ - probability</center>\n",
    "<center>$t$ - target</center>\n",
    "<center>$w$ - weight</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import neccessary packages\n",
    "from catboost import CatBoostClassifier, CatBoostRegressor, MultiTargetCustomMetric, MultiTargetCustomObjective\n",
    "import numpy as np\n",
    "from sklearn.datasets import make_classification, make_regression, make_multilabel_classification\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: for binary classification problems approxes are not equal to probabilities. Probabilities are calculated from approxes using sigmoid function.\n",
    "<h4><center>$p=\\frac{1}{1 + e^{-a}}=\\frac{e^a}{1 + e^a}$</center></h4>\n",
    "As an example, let's take Logloss metric which is defined by the following formula:\n",
    "<h4><center>$Logloss_i = -{w_i * (t_i * log(p_i) + (1 - t_i) * log(1 - p_i))}$</center></h4>\n",
    "<h4><center>$Logloss = \\frac{\\sum_{i=1}^{N}{Logloss_i}}{\\sum_{i=1}^{N}{w_i}}$</center></h4>\n",
    "This metric has derivative and can be used as objective. The derivatives of Logloss for single object are defined by the following formulas:\n",
    "<h4><center>$\\frac{\\partial(Logloss_i)}{\\partial a_i} = w_i * (t_i - p_i)$</center></h4>\n",
    "<h4><center>$\\frac{\\partial^2(Logloss_i)}{(\\partial a_i)^2} = -w_i * p_i * (1 - p_i)$</center></h4>\n",
    "Below you can see implemented Logloss objective and metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LoglossObjective(object):\n",
    "    def calc_ders_range(self, approxes, targets, weights):\n",
    "        assert len(approxes) == len(targets)\n",
    "        if weights is not None:\n",
    "            assert len(weights) == len(approxes)\n",
    "        \n",
    "        result = []\n",
    "        for index in range(len(targets)):\n",
    "            e = np.exp(approxes[index])\n",
    "            p = e / (1 + e)\n",
    "            der1 = targets[index] - p\n",
    "            der2 = -p * (1 - p)\n",
    "\n",
    "            if weights is not None:\n",
    "                der1 *= weights[index]\n",
    "                der2 *= weights[index]\n",
    "\n",
    "            result.append((der1, der2))\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LoglossMetric(object):\n",
    "    def get_final_error(self, error, weight):\n",
    "        return error / (weight + 1e-38)\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        return False\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        assert len(approxes) == 1\n",
    "        assert len(target) == len(approxes[0])\n",
    "\n",
    "        approx = approxes[0]\n",
    "\n",
    "        error_sum = 0.0\n",
    "        weight_sum = 0.0\n",
    "\n",
    "        for i in range(len(approx)):\n",
    "            e = np.exp(approx[i])\n",
    "            p = e / (1 + e)\n",
    "            w = 1.0 if weight is None else weight[i]\n",
    "            weight_sum += w\n",
    "            error_sum += -w * (target[i] * np.log(p) + (1 - target[i]) * np.log(1 - p))\n",
    "\n",
    "        return error_sum, weight_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below there are examples of training with built-in Logloss function and our Logloss objective and metric. As we can see, the results are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_classification(n_classes=2, random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.6900369\ttest: 0.6907175\tbest: 0.6907175 (0)\ttotal: 47.5ms\tremaining: 428ms\n",
      "1:\tlearn: 0.6866047\ttest: 0.6873479\tbest: 0.6873479 (1)\ttotal: 48.5ms\tremaining: 194ms\n",
      "2:\tlearn: 0.6835374\ttest: 0.6852325\tbest: 0.6852325 (2)\ttotal: 49.7ms\tremaining: 116ms\n",
      "3:\tlearn: 0.6804561\ttest: 0.6829075\tbest: 0.6829075 (3)\ttotal: 50.5ms\tremaining: 75.7ms\n",
      "4:\tlearn: 0.6776695\ttest: 0.6816999\tbest: 0.6816999 (4)\ttotal: 51.4ms\tremaining: 51.4ms\n",
      "5:\tlearn: 0.6749048\ttest: 0.6794533\tbest: 0.6794533 (5)\ttotal: 52.3ms\tremaining: 34.8ms\n",
      "6:\tlearn: 0.6712608\ttest: 0.6772634\tbest: 0.6772634 (6)\ttotal: 53ms\tremaining: 22.7ms\n",
      "7:\tlearn: 0.6681650\ttest: 0.6747041\tbest: 0.6747041 (7)\ttotal: 53.7ms\tremaining: 13.4ms\n",
      "8:\tlearn: 0.6658758\ttest: 0.6732683\tbest: 0.6732683 (8)\ttotal: 54.4ms\tremaining: 6.04ms\n",
      "9:\tlearn: 0.6633794\ttest: 0.6720979\tbest: 0.6720979 (9)\ttotal: 55ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6720978617\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54e5f9edd0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1 = CatBoostClassifier(iterations=10, loss_function='Logloss', eval_metric='Logloss',\n",
    "                            learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                            leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model1.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.6900380\ttest: 0.6907175\tbest: 0.6907175 (0)\ttotal: 447ms\tremaining: 4.02s\n",
      "1:\tlearn: 0.6866060\ttest: 0.6873479\tbest: 0.6873479 (1)\ttotal: 448ms\tremaining: 1.79s\n",
      "2:\tlearn: 0.6835392\ttest: 0.6852325\tbest: 0.6852325 (2)\ttotal: 449ms\tremaining: 1.05s\n",
      "3:\tlearn: 0.6804590\ttest: 0.6829075\tbest: 0.6829075 (3)\ttotal: 449ms\tremaining: 674ms\n",
      "4:\tlearn: 0.6776740\ttest: 0.6816999\tbest: 0.6816999 (4)\ttotal: 450ms\tremaining: 450ms\n",
      "5:\tlearn: 0.6749116\ttest: 0.6794533\tbest: 0.6794533 (5)\ttotal: 451ms\tremaining: 300ms\n",
      "6:\tlearn: 0.6712701\ttest: 0.6772634\tbest: 0.6772634 (6)\ttotal: 451ms\tremaining: 193ms\n",
      "7:\tlearn: 0.6681755\ttest: 0.6747041\tbest: 0.6747041 (7)\ttotal: 452ms\tremaining: 113ms\n",
      "8:\tlearn: 0.6658881\ttest: 0.6732683\tbest: 0.6732683 (8)\ttotal: 452ms\tremaining: 50.3ms\n",
      "9:\tlearn: 0.6633931\ttest: 0.6720979\tbest: 0.6720979 (9)\ttotal: 453ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6720978617\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54e5f7bad0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = CatBoostClassifier(iterations=10, loss_function=LoglossObjective(), eval_metric=LoglossMetric(), \n",
    "                            learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                            leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model2.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For regression approxes don't need any transformations. As an example of regression loss function and metric we take well-known RMSE which is defined by the following formulas:\n",
    "<h3><center>$RMSE = \\sqrt{\\frac{\\sum_{i=1}^{N}{w_i * (t_i - a_i)^2}}{\\sum_{i=1}^{N}{w_i}}}$</center></h3>\n",
    "It is more convenient to calculate MSE derivative, we will use it instead of RMSE derivative. It will not affect the solution as these metrics have the same optimums. \n",
    "<h4><center>$\\frac{\\partial(MSE)}{\\partial a_i} = w_i * (t_i - a_i)$</center></h4>\n",
    "<h4><center>$\\frac{\\partial^2(MSE)}{(\\partial a_i)^2} = -w_i$</center></h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RmseObjective(object):\n",
    "    def calc_ders_range(self, approxes, targets, weights):\n",
    "        assert len(approxes) == len(targets)\n",
    "        if weights is not None:\n",
    "            assert len(weights) == len(approxes)\n",
    "        \n",
    "        result = []\n",
    "        for index in range(len(targets)):\n",
    "            der1 = targets[index] - approxes[index]\n",
    "            der2 = -1\n",
    "\n",
    "            if weights is not None:\n",
    "                der1 *= weights[index]\n",
    "                der2 *= weights[index]\n",
    "\n",
    "            result.append((der1, der2))\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RmseMetric(object):\n",
    "    def get_final_error(self, error, weight):\n",
    "        return np.sqrt(error / (weight + 1e-38))\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        return False\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        assert len(approxes) == 1\n",
    "        assert len(target) == len(approxes[0])\n",
    "\n",
    "        approx = approxes[0]\n",
    "\n",
    "        error_sum = 0.0\n",
    "        weight_sum = 0.0\n",
    "\n",
    "        for i in range(len(approx)):\n",
    "            w = 1.0 if weight is None else weight[i]\n",
    "            weight_sum += w\n",
    "            error_sum += w * ((approx[i] - target[i])**2)\n",
    "\n",
    "        return error_sum, weight_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below there are examples of training with built-in RMSE function and our RMSE objective and metric. As we can see, the results are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_regression(random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 128.6631656\ttest: 140.6536718\tbest: 140.6536718 (0)\ttotal: 3.23ms\tremaining: 29ms\n",
      "1:\tlearn: 128.0351695\ttest: 140.7369887\tbest: 140.6536718 (0)\ttotal: 5.82ms\tremaining: 23.3ms\n",
      "2:\tlearn: 126.7781283\ttest: 141.0444768\tbest: 140.6536718 (0)\ttotal: 11.1ms\tremaining: 25.9ms\n",
      "3:\tlearn: 125.7603646\ttest: 141.1458855\tbest: 140.6536718 (0)\ttotal: 16.5ms\tremaining: 24.7ms\n",
      "4:\tlearn: 124.6922146\ttest: 141.0856002\tbest: 140.6536718 (0)\ttotal: 20.9ms\tremaining: 20.9ms\n",
      "5:\tlearn: 123.6667350\ttest: 141.0495141\tbest: 140.6536718 (0)\ttotal: 23.3ms\tremaining: 15.5ms\n",
      "6:\tlearn: 122.7210914\ttest: 140.8511986\tbest: 140.6536718 (0)\ttotal: 30.7ms\tremaining: 13.2ms\n",
      "7:\tlearn: 121.8418528\ttest: 140.7646996\tbest: 140.6536718 (0)\ttotal: 35.5ms\tremaining: 8.88ms\n",
      "8:\tlearn: 121.0103984\ttest: 140.4834561\tbest: 140.4834561 (8)\ttotal: 39.4ms\tremaining: 4.38ms\n",
      "9:\tlearn: 119.9286951\ttest: 140.2935285\tbest: 140.2935285 (9)\ttotal: 43.7ms\tremaining: 0us\n",
      "\n",
      "bestTest = 140.2935285\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7f55340dce50>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1 = CatBoostRegressor(iterations=10, loss_function='RMSE', eval_metric='RMSE',\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model1.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 128.6631656\ttest: 140.6536718\tbest: 140.6536718 (0)\ttotal: 410ms\tremaining: 3.69s\n",
      "1:\tlearn: 128.0351695\ttest: 140.7369887\tbest: 140.6536718 (0)\ttotal: 413ms\tremaining: 1.65s\n",
      "2:\tlearn: 126.7781283\ttest: 141.0444768\tbest: 140.6536718 (0)\ttotal: 416ms\tremaining: 971ms\n",
      "3:\tlearn: 125.7603646\ttest: 141.1458855\tbest: 140.6536718 (0)\ttotal: 418ms\tremaining: 627ms\n",
      "4:\tlearn: 124.6922146\ttest: 141.0856002\tbest: 140.6536718 (0)\ttotal: 420ms\tremaining: 420ms\n",
      "5:\tlearn: 123.6667350\ttest: 141.0495141\tbest: 140.6536718 (0)\ttotal: 422ms\tremaining: 282ms\n",
      "6:\tlearn: 122.7210914\ttest: 140.8511986\tbest: 140.6536718 (0)\ttotal: 424ms\tremaining: 182ms\n",
      "7:\tlearn: 121.8418528\ttest: 140.7646996\tbest: 140.6536718 (0)\ttotal: 426ms\tremaining: 106ms\n",
      "8:\tlearn: 121.0103984\ttest: 140.4834561\tbest: 140.4834561 (8)\ttotal: 427ms\tremaining: 47.5ms\n",
      "9:\tlearn: 119.9286951\ttest: 140.2935285\tbest: 140.2935285 (9)\ttotal: 429ms\tremaining: 0us\n",
      "\n",
      "bestTest = 140.2935285\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7f54e5f9e990>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = CatBoostRegressor(iterations=10, loss_function=RmseObjective(), eval_metric=RmseMetric(),\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model2.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Multiclassification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: for multiclassification problems approxes are not equal to probabilities. Usually approxes are transformed to probabilities using Softmax function.\n",
    "<h3><center>$p_{i,c} = \\frac{e^{a_{i,c}}}{\\sum_{j=1}^k{e^{a_{i,j}}}}$</center></h3>\n",
    "<center>$p_{i,c}$ - the probability that $x_i$ belongs to class $c$</center>\n",
    "<center>$k$ - number of classes</center>\n",
    "<center>$a_{i,j}$ - approx for object $x_i$ for class $j$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement MultiClass objective that is defined as follows:\n",
    "<h3><center>$MultiClass_i = w_i * \\log{p_{i,t_i}}$</center></h3>\n",
    "<h3><center>$MultiClass = \\frac{\\sum_{i=1}^{N}Multiclass_i}{\\sum_{i=1}^{N}w_i}$</center></h3>\n",
    "\n",
    "<h3><center>$\\frac{\\partial(Multiclass_i)}{\\partial{a_{i,c}}} = \\begin{cases} \n",
    "w_i-\\frac{w_i*e^{a_{i,c}}}{\\sum_{j=1}^{k}e^{a_{i,j}}}, & \\mbox{if } c = t_i \\\\ \n",
    "-\\frac{w_i*e^{a_{i,c}}}{\\sum_{j=1}^{k}e^{a_{i,j}}}, & \\mbox{if } c \\neq t_i \n",
    "\\end{cases}$</center></h3>\n",
    "\n",
    "<h3><center>$\\frac{\\partial^2(Multiclass_i)}{\\partial{a_{i,c_1}}\\partial{a_{i,c_2}}} = \\begin{cases} \n",
    "\\frac{w_i*e^{2*a_{i,c_1}}}{(\\sum_{j=1}^{k}e^{a_{i,j}})^2} - \\frac{w_i*e^{a_{i, c_1}}}{\\sum_{j=1}^{k}e^{a_{i,j}}}, & \\mbox{if } c_1 = c_2 \\\\ \n",
    "\\frac{w_i*e^{a_{i,c_1}+a_{i,c_2}}}{(\\sum_{j=1}^{k}e^{a_{i,j}})^2}, & \\mbox{if } c_1 \\neq c_2 \n",
    "\\end{cases}$</center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiClassObjective(object):\n",
    "    def calc_ders_multi(self, approx, target, weight):\n",
    "        approx = [approx[i] - max(approx) for i in range(len(approx))]\n",
    "        exp_approx = [np.exp(approx[i]) for i in range(len(approx))]\n",
    "        exp_sum = 0.0\n",
    "        for v in exp_approx:\n",
    "            exp_sum += v\n",
    "        grad = []\n",
    "        hess = []\n",
    "        for j in range(len(approx)):\n",
    "            der1 = -exp_approx[j] / exp_sum\n",
    "            if j == target:\n",
    "                der1 += 1\n",
    "            hess_row = []\n",
    "            for j2 in range(len(approx)):\n",
    "                der2 = exp_approx[j] * exp_approx[j2] / (exp_sum**2)\n",
    "                if j2 == j:\n",
    "                    der2 -= exp_approx[j] / exp_sum\n",
    "                hess_row.append(der2 * weight)\n",
    "                \n",
    "            grad.append(der1 * weight)\n",
    "            hess.append(hess_row)\n",
    "            \n",
    "        return (grad, hess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AccuracyMetric(object):\n",
    "    def get_final_error(self, error, weight):\n",
    "        return error / (weight + 1e-38)\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        return True\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        best_class = np.zeros(len(approxes[0]))\n",
    "        \n",
    "        for i in range(len(approxes[0])):\n",
    "            approx_i = [approxes[j][i] for j in range(len(approxes))]\n",
    "            best_class[i] = np.argmax(np.array(approx_i))\n",
    "        \n",
    "        accuracy_sum = 0\n",
    "        weight_sum = 0 \n",
    "\n",
    "        for i in range(len(target)):\n",
    "            w = 1.0 if weight is None else weight[i]\n",
    "            weight_sum += w\n",
    "            \n",
    "            accuracy_sum += w * (best_class[i] == target[i])\n",
    "\n",
    "        return accuracy_sum, weight_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below there are examples of training with built-in MultiClass function and our MultiClass objective. As we can see, the results are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_classification(n_samples=1000, n_features=50, n_informative=40, n_classes=5, random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.3893333\ttest: 0.2200000\tbest: 0.2200000 (0)\ttotal: 25.9ms\tremaining: 233ms\n",
      "1:\tlearn: 0.4933333\ttest: 0.2680000\tbest: 0.2680000 (1)\ttotal: 41.9ms\tremaining: 167ms\n",
      "2:\tlearn: 0.5493333\ttest: 0.3240000\tbest: 0.3240000 (2)\ttotal: 56.8ms\tremaining: 132ms\n",
      "3:\tlearn: 0.6080000\ttest: 0.3200000\tbest: 0.3240000 (2)\ttotal: 72.8ms\tremaining: 109ms\n",
      "4:\tlearn: 0.6640000\ttest: 0.3240000\tbest: 0.3240000 (2)\ttotal: 86ms\tremaining: 86ms\n",
      "5:\tlearn: 0.6880000\ttest: 0.3720000\tbest: 0.3720000 (5)\ttotal: 99.3ms\tremaining: 66.2ms\n",
      "6:\tlearn: 0.7106667\ttest: 0.3640000\tbest: 0.3720000 (5)\ttotal: 113ms\tremaining: 48.3ms\n",
      "7:\tlearn: 0.7066667\ttest: 0.3440000\tbest: 0.3720000 (5)\ttotal: 127ms\tremaining: 31.8ms\n",
      "8:\tlearn: 0.7320000\ttest: 0.3800000\tbest: 0.3800000 (8)\ttotal: 140ms\tremaining: 15.6ms\n",
      "9:\tlearn: 0.7453333\ttest: 0.3800000\tbest: 0.3800000 (8)\ttotal: 156ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.38\n",
      "bestIteration = 8\n",
      "\n",
      "Shrink model to first 9 iterations.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54da793dd0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1 = CatBoostClassifier(iterations=10, loss_function='MultiClass', eval_metric='Accuracy',\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Newton', classes_count=5)\n",
    "model1.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.3893333\ttest: 0.2200000\tbest: 0.2200000 (0)\ttotal: 949ms\tremaining: 8.54s\n",
      "1:\tlearn: 0.4933333\ttest: 0.2680000\tbest: 0.2680000 (1)\ttotal: 969ms\tremaining: 3.88s\n",
      "2:\tlearn: 0.5493333\ttest: 0.3240000\tbest: 0.3240000 (2)\ttotal: 988ms\tremaining: 2.31s\n",
      "3:\tlearn: 0.6080000\ttest: 0.3200000\tbest: 0.3240000 (2)\ttotal: 1.01s\tremaining: 1.51s\n",
      "4:\tlearn: 0.6640000\ttest: 0.3240000\tbest: 0.3240000 (2)\ttotal: 1.03s\tremaining: 1.03s\n",
      "5:\tlearn: 0.6880000\ttest: 0.3720000\tbest: 0.3720000 (5)\ttotal: 1.04s\tremaining: 697ms\n",
      "6:\tlearn: 0.7106667\ttest: 0.3640000\tbest: 0.3720000 (5)\ttotal: 1.06s\tremaining: 456ms\n",
      "7:\tlearn: 0.7066667\ttest: 0.3440000\tbest: 0.3720000 (5)\ttotal: 1.08s\tremaining: 271ms\n",
      "8:\tlearn: 0.7320000\ttest: 0.3800000\tbest: 0.3800000 (8)\ttotal: 1.1s\tremaining: 123ms\n",
      "9:\tlearn: 0.7453333\ttest: 0.3800000\tbest: 0.3800000 (8)\ttotal: 1.13s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.38\n",
      "bestIteration = 8\n",
      "\n",
      "Shrink model to first 9 iterations.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54da788210>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = CatBoostClassifier(iterations=10, loss_function=MultiClassObjective(), eval_metric=AccuracyMetric(),\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Newton', classes_count=5)\n",
    "model2.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Multiregression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "Multi-target custom objectives and metrics are required to be inherited from <code>cb.MultiTargetCustomObjective</code> and <code>cb.MultiTargetCustomMetric</code> respectively.\n",
    "</div>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will take MultiRMSE as an example for multiregression loss function and metric:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3><center>$MultiRMSE = \\sqrt{\\displaystyle\\frac{\\sum_{i=1}^{N}\\sum_{d=1}^{dim}(a_{i,d} - t_{i, d})^{2} w_{i}}{\\sum_{i=1}^{N} w_{i}}},$</center></h3>\n",
    "$dim$ is the identifier of the dimension of the target. Here we also will use MultiMSE derivatives as it is more convinient to calculate.\n",
    "<h3><center>$\\frac{\\partial(MultiMSE)}{\\partial(a_{i,d})} = w_i * (t_i - a_i)$</center></h3>\n",
    "<h3><center>$\\frac{\\partial^2(MultiMSE)}{(\\partial a_{i,d})^2} = -w_i$</center></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiRmseObjective(MultiTargetCustomObjective):\n",
    "    def calc_ders_multi(self, approx, target, weight):\n",
    "        assert len(target) == len(approx)\n",
    "        \n",
    "        w = weight if weight is not None else 1.0\n",
    "        der1 = [(target[i] - approx[i]) * w for i in range(len(approx))]\n",
    "        der2 = [-w for i in range(len(approx))]\n",
    "\n",
    "        return (der1, der2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiRmseMetric(MultiTargetCustomMetric):\n",
    "    def get_final_error(self, error, weight):\n",
    "        return np.sqrt(error / (weight + 1e-38))\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        return False\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        assert len(target) == len(approxes)\n",
    "        assert len(target[0]) == len(approxes[0])\n",
    "\n",
    "        error_sum = 0.0\n",
    "        weight_sum = 0.0\n",
    "        \n",
    "        for i in range(len(approxes[0])):\n",
    "            w = 1.0 if weight is None else weight[i]\n",
    "            weight_sum += w\n",
    "            for d in range(len(approxes)):\n",
    "                error_sum += w * ((approxes[d][i] - target[d][i])**2)\n",
    "        return error_sum, weight_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below there are examples of training with built-in MultiRMSE function and our MultiRMSE objective. As we can see, the results are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_regression(random_state=0, n_targets=3)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 259.1763401\ttest: 282.0142823\tbest: 282.0142823 (0)\ttotal: 9.37ms\tremaining: 84.3ms\n",
      "1:\tlearn: 257.6741584\ttest: 281.6794060\tbest: 281.6794060 (1)\ttotal: 16.5ms\tremaining: 66ms\n",
      "2:\tlearn: 256.0533781\ttest: 281.6770678\tbest: 281.6770678 (2)\ttotal: 21.8ms\tremaining: 50.9ms\n",
      "3:\tlearn: 254.5986664\ttest: 282.3533075\tbest: 281.6770678 (2)\ttotal: 26.5ms\tremaining: 39.8ms\n",
      "4:\tlearn: 252.7508961\ttest: 281.5696382\tbest: 281.5696382 (4)\ttotal: 30.9ms\tremaining: 30.9ms\n",
      "5:\tlearn: 250.7948713\ttest: 281.0709409\tbest: 281.0709409 (5)\ttotal: 35.6ms\tremaining: 23.7ms\n",
      "6:\tlearn: 249.8911839\ttest: 281.4832428\tbest: 281.0709409 (5)\ttotal: 43.9ms\tremaining: 18.8ms\n",
      "7:\tlearn: 248.3460853\ttest: 282.1097335\tbest: 281.0709409 (5)\ttotal: 50.3ms\tremaining: 12.6ms\n",
      "8:\tlearn: 246.9427803\ttest: 281.7479127\tbest: 281.0709409 (5)\ttotal: 54.8ms\tremaining: 6.09ms\n",
      "9:\tlearn: 245.7896028\ttest: 282.3497824\tbest: 281.0709409 (5)\ttotal: 60.3ms\tremaining: 0us\n",
      "\n",
      "bestTest = 281.0709409\n",
      "bestIteration = 5\n",
      "\n",
      "Shrink model to first 6 iterations.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7f54e10d7890>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1 = CatBoostRegressor(iterations=10, loss_function='MultiRMSE', eval_metric='MultiRMSE',\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model1.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 259.1763401\ttest: 282.0142823\tbest: 282.0142823 (0)\ttotal: 493ms\tremaining: 4.44s\n",
      "1:\tlearn: 257.6741584\ttest: 281.6794060\tbest: 281.6794060 (1)\ttotal: 500ms\tremaining: 2s\n",
      "2:\tlearn: 256.0533781\ttest: 281.6770678\tbest: 281.6770678 (2)\ttotal: 505ms\tremaining: 1.18s\n",
      "3:\tlearn: 254.5986664\ttest: 282.3533075\tbest: 281.6770678 (2)\ttotal: 510ms\tremaining: 765ms\n",
      "4:\tlearn: 252.7508961\ttest: 281.5696382\tbest: 281.5696382 (4)\ttotal: 515ms\tremaining: 515ms\n",
      "5:\tlearn: 250.7948713\ttest: 281.0709409\tbest: 281.0709409 (5)\ttotal: 521ms\tremaining: 347ms\n",
      "6:\tlearn: 249.8911839\ttest: 281.4832428\tbest: 281.0709409 (5)\ttotal: 528ms\tremaining: 226ms\n",
      "7:\tlearn: 248.3460853\ttest: 282.1097335\tbest: 281.0709409 (5)\ttotal: 535ms\tremaining: 134ms\n",
      "8:\tlearn: 246.9427803\ttest: 281.7479127\tbest: 281.0709409 (5)\ttotal: 541ms\tremaining: 60.1ms\n",
      "9:\tlearn: 245.7896028\ttest: 282.3497824\tbest: 281.0709409 (5)\ttotal: 545ms\tremaining: 0us\n",
      "\n",
      "bestTest = 281.0709409\n",
      "bestIteration = 5\n",
      "\n",
      "Shrink model to first 6 iterations.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7f54da788f50>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = CatBoostRegressor(iterations=10, loss_function=MultiRmseObjective(), eval_metric=MultiRmseMetric(),\n",
    "                           learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                           leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model2.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. MultiLabel Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "Multi-target custom objectives and metrics are required to be inherited from <code>catboost.MultiTargetCustomObjective</code> and <code>catboost.MultiTargetCustomMetric</code> respectively.\n",
    "</div>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: for binary multilabel classification problems approxes are not equal to probabilities. Probability for $i$ object and $j$ class dimension is calculated from approxes using sigmoid function:\n",
    "<h4><center>$p_{ij}=\\frac{1}{1 + e^{-a_{ij}}}=\\frac{e^{a_{ij}}}{1 + e^{a_{ij}}}$</center></h4>\n",
    "As an example, let's take MultiLogloss metric which is defined by the following formula:\n",
    "<h3><center>$MultiLogloss = \\frac{-\\sum_{j=0}^{M-1} \\sum_{i=1}^{N} w_{i} (t_{ij} \\log p_{ij} + (1-t_{ij}) \\log (1 - p_{ij}) )}{M\\cdot\\sum_{i=1}^{N}w_{i}} { ,}$</center></h3>\n",
    "\n",
    "  where $t_{ij} \\in {0, 1}$ is the corresponding class.  \n",
    "This metric has derivative and can be used as objective. The derivatives of MultiLogloss for $i$ object and $j$ class dimension are defined by the following formulas:\n",
    "<h4><center>$\\frac{\\partial(MultiLogloss)}{\\partial(a_{ij})} = w_i * (t_{ij} - p_{ij})$</center></h4>\n",
    "<h4><center>$\\frac{\\partial^2(MultiLogloss)}{(\\partial a_{ij})^2} = -w_i * p_{ij} * (1 - p_{ij})$</center></h4>\n",
    "Below you can see implemented MultiLogloss objective and metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiLoglossObjective(MultiTargetCustomObjective):\n",
    "    def calc_ders_multi(self, approx, target, weight):\n",
    "        assert len(target) == len(approx)\n",
    "        \n",
    "        e = np.exp(approx)\n",
    "        p = e / (1 + e)\n",
    "        \n",
    "        w = weight if weight is not None else 1.0\n",
    "        der1 = [(target[i] - p[i]) * w for i in range(len(approx))]\n",
    "        der2 = [-p[i] * (1 - p[i]) * w for i in range(len(approx))]\n",
    "\n",
    "        return (der1, der2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiLoglossMetric(MultiTargetCustomMetric):\n",
    "    def get_final_error(self, error, weight):\n",
    "        return error / (weight + 1e-38)\n",
    "\n",
    "    def is_max_optimal(self):\n",
    "        return False\n",
    "\n",
    "    def evaluate(self, approxes, target, weight):\n",
    "        assert len(target) == len(approxes)\n",
    "        assert len(target[0]) == len(approxes[0])\n",
    "\n",
    "        error_sum = 0.0\n",
    "        weight_sum = 0.0\n",
    "\n",
    "        for i in range(len(approxes[0])):\n",
    "            w = 1.0 if weight is None else weight[i]\n",
    "            weight_sum += w\n",
    "            for d in range(len(approxes)):\n",
    "                e = np.exp(approxes[d][i])\n",
    "                p = e / (1 + e)\n",
    "                error_sum += -w * (target[d][i] * np.log(p) + (1 - target[d][i]) * np.log(1 - p))\n",
    "        return error_sum / len(approxes), weight_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below there are examples of training with built-in MultiLogloss function and our MultiLogloss objective. As we can see, the results are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_multilabel_classification(random_state=0, n_classes=3)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.6913620\ttest: 0.6923958\tbest: 0.6923958 (0)\ttotal: 645us\tremaining: 5.81ms\n",
      "1:\tlearn: 0.6893857\ttest: 0.6919275\tbest: 0.6919275 (1)\ttotal: 1.35ms\tremaining: 5.39ms\n",
      "2:\tlearn: 0.6874246\ttest: 0.6909644\tbest: 0.6909644 (2)\ttotal: 2.31ms\tremaining: 5.38ms\n",
      "3:\tlearn: 0.6854196\ttest: 0.6897256\tbest: 0.6897256 (3)\ttotal: 2.79ms\tremaining: 4.18ms\n",
      "4:\tlearn: 0.6832049\ttest: 0.6888271\tbest: 0.6888271 (4)\ttotal: 3.49ms\tremaining: 3.49ms\n",
      "5:\tlearn: 0.6810096\ttest: 0.6885706\tbest: 0.6885706 (5)\ttotal: 3.99ms\tremaining: 2.66ms\n",
      "6:\tlearn: 0.6790223\ttest: 0.6875159\tbest: 0.6875159 (6)\ttotal: 4.76ms\tremaining: 2.04ms\n",
      "7:\tlearn: 0.6773518\ttest: 0.6866457\tbest: 0.6866457 (7)\ttotal: 5.57ms\tremaining: 1.39ms\n",
      "8:\tlearn: 0.6750769\ttest: 0.6850851\tbest: 0.6850851 (8)\ttotal: 6.07ms\tremaining: 674us\n",
      "9:\tlearn: 0.6731996\ttest: 0.6845872\tbest: 0.6845872 (9)\ttotal: 6.55ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6845871591\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54e10d7450>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1 = CatBoostClassifier(iterations=10, loss_function='MultiLogloss', eval_metric='MultiLogloss',\n",
    "                            learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                            leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model1.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.6913620\ttest: 0.6923958\tbest: 0.6923958 (0)\ttotal: 547ms\tremaining: 4.92s\n",
      "1:\tlearn: 0.6893857\ttest: 0.6919275\tbest: 0.6919275 (1)\ttotal: 548ms\tremaining: 2.19s\n",
      "2:\tlearn: 0.6874246\ttest: 0.6909644\tbest: 0.6909644 (2)\ttotal: 549ms\tremaining: 1.28s\n",
      "3:\tlearn: 0.6854196\ttest: 0.6897256\tbest: 0.6897256 (3)\ttotal: 549ms\tremaining: 824ms\n",
      "4:\tlearn: 0.6832049\ttest: 0.6888271\tbest: 0.6888271 (4)\ttotal: 550ms\tremaining: 550ms\n",
      "5:\tlearn: 0.6810096\ttest: 0.6885706\tbest: 0.6885706 (5)\ttotal: 550ms\tremaining: 367ms\n",
      "6:\tlearn: 0.6790223\ttest: 0.6875159\tbest: 0.6875159 (6)\ttotal: 551ms\tremaining: 236ms\n",
      "7:\tlearn: 0.6773518\ttest: 0.6866457\tbest: 0.6866457 (7)\ttotal: 551ms\tremaining: 138ms\n",
      "8:\tlearn: 0.6750769\ttest: 0.6850851\tbest: 0.6850851 (8)\ttotal: 552ms\tremaining: 61.3ms\n",
      "9:\tlearn: 0.6731996\ttest: 0.6845872\tbest: 0.6845872 (9)\ttotal: 552ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6845871591\n",
      "bestIteration = 9\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostClassifier at 0x7f54e10d7d90>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2 = CatBoostClassifier(iterations=10, loss_function=MultiLoglossObjective(), eval_metric=MultiLoglossMetric(),\n",
    "                            learning_rate=0.03, bootstrap_type='Bayesian', boost_from_average=False,\n",
    "                            leaf_estimation_iterations=1, leaf_estimation_method='Gradient')\n",
    "model2.fit(X_train, y_train, eval_set=(X_test, y_test))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
